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INTEGRATED VIRTUAL SIDE AND LIVE MICROSCOPE SYSTEM 



BACKGROUND OF THE INVENTION 

[001] This invention relates to a method and apparatus for viewing remote 

microscope images. 

[002] Currently there is increasing demand for pathologist review of samples at 

remote locations. There exist multiple systems to address this need. They typically fall 
within one of two categories: live remote microscopy and virtual slide imaging. 

[003] In live remote control microscopy, a user receives images that are taken 

from a slide on a microscope. In virtual slide imaging, a user receives images previously 
captured. Virtual slide systems take one or more images of an area of interest and 
assemble them together. (if there is more than one image) to form a virtual slide. Each of 
these techniques has its advantages. Live remote imaging provides users with the closest 
approximation to manual manipulation. Virtual slides allow faster image viewing, since 
images are already captured. 

[004] Virtual slide systems take one or more images and assemble them to form a 

"virtual slide." 

[005] However, users in the past were limited in their ability to integrate these 

technologies. One could only view and manipulate live and virtual images independently 
of one another. A user would have to clumsily go back and forth between these two 
modes of operation to separately look at the virtual slides and live microscope slides. We 
present a new method that integrates these ideas into one seamless operating 
environment. 
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SUMMARY OF THE INVENTION 



[006] In accordance with the present invention, a method and apparatus are 

provided for the analysis of remote slides in a hybrid live and virtual medium. Users 
obtain benefits of each technique in a unified environment. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[007] A virtual slide is a digital representation of an area of interest of a 

microscopic slide. A virtual slide can be created multiple ways. 

; [008] One method is to use a motorized microscope that can move a specimen 

with respect to a microscope objective (e.g. , a microscope with a motorized stage). With 
such system, one can capture one or more images through a microscope objective, such 
that a region of interest (all or part of the microscopic slide) is imaged. Each image is 
then joined together to form a composite or "virtual image." Multiple methods of joining 
images together are known in the art. One example is when images are simply abutted 
one next to another. However, this method does not generally produce virtual slides 
without seams, because errors such as camera rotation relative to the axis of motion are 
difficult to correct. Even with submicron accuracy stages it is, in practice, difficult to 
obtain consistent positioning. Another method is to utilize overlap between adjacent 
images to edge align images for maximum seamlessness. This can be done by 
sequentially shifting overlapping regions in the x and/or y axis, for example, by a 
stepping motor, and calculating a correlation value (or measure of goodness of overlap). 
The shift which results in the best correlation value is then used to join the images 
together (Figure 1). While this method can be computationally expensive, it reduces 
reliance on difficult-to-attain mechanical positioning requirements and ultimately 
produces the best images in the sense of seamlessness. 
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[009] In another method, the virtual slide is made simply by utilizing an imaging 

device with optics suitable to take a an image of the area of interest on the slide in one 
snapshot. This method is embodied in the form of a conventional digital or analog 
camera with a macro lens. 

[010] This virtual slide can then be used to create a thumbnail view of the slide. 

To create the thumbnail view, the virtual slide is shrunk in resolution from its original, 
base resolution to a target resolution. If the target resolution is the same as the base 
resolution, then the image is unchanged. However, typically the resolution of the 
thumbnail desired is several times smaller than the base resolution. 

[Oil] With the virtual slide created, the user may fully utilize the full capabilities 

of the remote microscope. The user is presented with an image window and a set of 
control features (Figure 2). Among these control features is a set of "optical objectives" 
and "virtual objectives." 

[012] Optical objectives are images that are created by a camera digitizing an 

image through a microscope objective (e.g., lOx, 20x, or 40x) in real time (re:, an image 
is captured at the time the user requests the image). Virtual objectives are digitally ; 
created magnifications created not by digitizing in real time, but rather by utilizing the 
existing virtual slide data to digitally create a field of view. 

[013] When a user selects one of the optical objectives, a "change objective" 

command is sent to the microscope. This change objective command can also specify 
additional qualifying information, such as microscope x, y, z positions, exposure setting, 
compression type and level, and image dimensions. If additional qualifying information 
is not sent, then the implicit qualify ing information is the current state of the microscope 
or the last specified state. When the microscope receives the command, actions are taken 
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to change the objective lens and to change the state of the microscope commensurate with 
the command (e.g., change the relative position of the objective lens relative to the 
microscopic slide, change exposure, etc.). An image is then digitized, compressed if so 
specified, and then transmitted to the user for display. 

[014] When a user selects a virtual objective, a virtual objective command is sentj 

to the microscope. Similar to an optical objective command, this virtual objective 
command can also specify additional qualifying information, such as microscope x, y, 
and/or z position. If additional qualifying information is not sent, then the implicit 
qualifying information is current microscope state or the last specified state. A region of 
interest is defined by the virtual request - it is the area on the microscope slide included 
in the field specified by the coordinates of the stage x, y and magnification of the 
command. This region of interest may optionally be trimmed such that image 
information already residing at the requesting user's view is not retransmitted to the user. 

[015] An image of the region of interest can be created from the virtual slide in 

.multiple ways. If the virtual slide is not compressed, then retrieval of image information 
corresponding to the region of interest can be done by simply copying data from the - 
virtual slide. If the virtual slide is compressed, a region corresponding to at least the : 
region of interest can be decompressed to a raw bitmap from the main compressed image . 
If the virtual slide was stored as multiple compressed images rather than one large image, 
additional efficiencies are possible. For example, only those images that contain the 
desired region of interest need be accessed for decompression, rather than the entire area 
of the virtual slide. This enhances performance. 

[016] The decompression itself can also be performed in various ways. Strategies 

such as scaled decoding, as in the case of jpeg type compression, can be employed to 
improve the speed of the decompression by coupling a resolution reduction process with 
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the decompression to speed up decompression when resolution reduction is required. 
Once the region of interest is decompressed, it can then be recompressed using a variety 
of strategies known to those skilled in the imaging field, which need not be the same as 
the method by which the virtual slide was compressed., 

[017] An alternative type of decompression/recompression step can also be used 

involving partial decompression. Partial decompression, such as decoding of Huffman- 
encoded data, as in the case of jpeg, can be performed to produce raw coefficients, rather 
than full decompression, which produces a raw bitmap. The raw coefficients 
corresponding to the area of the region of interest can then be selected. These 
coefficients which correspond to the region of interest are then re-encoded. In the case of 
jpeg compression, this would involve re Huffman encoding of the coefficients, rather than 
in normal full compression, where a dct must be performed followed by quantization and 
then Huffman coding. 

[018] Whichever the technique, the result is a compressed region of interest. The 

compressed region of interest is then transmitted to the user for viewing. The described 
method is more advantageous than sending the entire virtual slide, as one efficiently 
sends only that information required by the user. 

[019] However, direct transfer of the compressed image without decompression is 

feasible when the virtual slide is stored as multiple compressed images. The compressed 
images that include the area specified by the region of interest can be directly transferred 
to the user, rather than going through a decompression/recompression step. The 
disadvantage is that one may transfer more information than is needed if, for example, the 
compressed images are at a higher resolution that the requested resolution. This can be 
partially solved by creation of multiple resolution versions of the virtual slide. There are 
also compression strategies available that allow only portions of the compressed images 
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to be sent, such that a given resolution can be attained depending on which portions of 
the compressed image one chooses to send (e.g., progressive encoding). However, there 
is still the issue that the region of interest only partially covers the area of the compressed 
image. In this case, direct transfer of the image results in inefficiently sending data 
including both the region of interest and data outside the region of interest to the user. 

[020] With this invention, users are afforded a streamlined method of utilizing the 

features of virtual and live microscopy techniques. 

) 



BRIEF DESCRIPTION OF THE DRAWINGS 

[022) Figure 1 is a drawing showing the overlap between adjacent images during 

optimization of overlap. 

[023] Figure 2 is a photograph of a user's view of the remote microscope, 

showing a thumbnail view, microscope imaging window, and a set of microscope 
controls. 

[024] While the apparatus and methods of the present invention have been 

illustrated in terms of certain embodiments, the invention claimed herein is not limited to 
the embodiments disclosed in this application. Rather, the scope of the invention is 
defined by the claims attached hereto. 
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